Academic Data Collection in Electronic Environments: Defining Acceptable Use of Internet Resources

The Internet contains data in a variety of forms in a large number of websites. The content of websites and interactions through websites are the subject of academic research. Research about web content and web interactions can be facilitated through the use of automated Internet data collection agents. The unique characteristics of the Internet and the new capabilities of the data collection agents raise questions about the propriety and legality of this method of research. Automated data collection agents can be compared to resource discovery robots (bots) that arbitrarily crawl the web to identify content for Internet resource indexing. Automated data collection agents target specific sites for pre-defined data to be analyzed in academic research. Websites are maintained by owners in order to facilitate commerce or present information to be read by Internet users. Automated access to a site’s resources causes the site to incur some cost, without participating in the intended goal of the site. Accordingly, automated requests for data often undesired and sometimes blocked by site administration. However, academic research conducted in this manner results in a desirable public good. Therefore, as long as the cost of academic access to selected websites is small and does not constitute an undue burden, it should be allowed—perhaps encouraged. Two major legal challenges to the use of automated data collection agents for academic research use are based on the legal doctrines of trespass and copyright. The legal theory of trespass to chattels has been used successfully to enjoin individuals from conducting automated data collection. Additionally, copies of pages requested from websites are routinely archived in the data collection process may constitute a violation of copyright. Although the academic activities of teaching and research are preferred activities under U.S. copyright law’s “fair use” exclusion, most countries do not have similar provisions favoring academic use. To alleviate legal concerns, a researcher may seek the express consent of a website’s administration to conduct automated data retrieval; however, this may be undesirable for two reasons. First, significant costs are associated with the approval process and second, when a site knows that it is being accessed for research, the site managers may change the behavior of the site. In some cases, general express consent is provided when the site posts a robots.txt file specifying which areas of the site may be accessed by automated means. When general express consent is not indicated, sites may indicate implied consent simply by their connection to a public network in which automated retrieval mechanisms are in common use. Guidelines for researchers are proposed to reduce objections to research activities and to facilitate communication with website administration. These include notification to website administration of intended automated research activity, description of the research project, and clear identification of each automated request so site administration can clearly understand the impact of the activity on site performance.